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Brandt, 'Ronald. S,, Editor. Applied Strategies for 
Curricultjm' Eyalualion. Alexandria/ Virginia:" Asso- . 
elation for Supervision and Curriculum Develop- 
ment, 1^81 .'130 paces'. ED 211 582. 

A school's humanities* program ji? probably the most difficult 
part of the instructional prograrato evaluate. )n fact, states one of 
the contributing authors of this refreshing, and sometimes humor- 
ous, publication, "Evaluating a humanities program holds chal- 
lenges akin to those involved in evaluating a formal religion": hu- 
manities educators, with their "unbridled enthusiasm and lofty 
aspirations for their programs," are incredi/lous that ariyone would 
even attempt to evaluate their high-minded programs. ' 

But in this era of accountability^ and budget cuts, even hu- 
manities programs must justify their existence. This excellent publi- 
cation shows how humanities program evaluations mi'ght' be car- 
ried out. Editor Brandt asked eight experts in the field of educa- 
tional evaluation hqw they woufd evaluate a specific humanities 
progfam— that af Radnor (Pennsylvania) fiddle School. Their re,-- 
sponses— compiled into seven chapter^— provide a^ wealth of 
useful Ideas and methods, most of which are applicable beyond 
humanities programs to all instructional programs. 

For example, Deborah G. Bonnett— an experienced practitioner 
\Af1th wide experience in educational evaluation— outlines* the 
ejfiht-step process she would use to evaluate the program. William 



e^i 
We 



5ebst^r,-a fbll-time evaluator in the D^fllas (Texas) Independent 
School District, describes the CIPP (Context, Input, Process, Pro- 
duct) 



model 

winning pioneer of adversary evaluation 



he would use. And Blaine Worthen— an award- 
explains Jn detail his 

assessment^of the humanities program in a chapter titled "Journal 
Entries of an Eclectic Evaluator." Included are two appendixes out- 
lining the history of Radnor's humanities program and a. report 
from the district's humanities curriculum review^committee. 

(sj^ Caiifornia School Boards Association. How to 

Evaluate Your School instructional Program. Curr/cu-^. 
lum is a Board Member Responsibility. Sacramento: 
CSBA, November 1981. 36 pages. ED 214 232. 

New school board members are often enthused about im- 
proving their district's instructional program. Once thev are seated 

• at the board table, however, they begin to realize how complex and 

• numerous curriculum issues are. A series of rationalizations for 
* noninvolvement in program evaluation often follows, including the 

^ ' ' familiar' "Our probFem is. money— not-curriculum." 
g|^l^3ut involvement in curriculum improvement is in fact a legal ^ 



school board responsibility, as chapter. Kof this excellent publi- 
cation shows. Jhis involverpientvdoes not mean a "take-over" of 
superintendent *responsi bill ties: "Jhe ablest and rqpst heJpful , 
school board members," the autf^ors point out, "are th^ose who 
become I^nowledgeable about and keenly intojested in the curri- 
culum, tjut who are willing to leav^cfetailed and technical school 
management and all instructional^^ters to the superinWi'dent 
and his or her staff." 

The second chapter discusses how board members can become 
"comfortable" with curriculum involverftent. First- steps include 
reading, workshops, 'and classroom visits. Practical advice is given 
to make classroom visitation successful, including questions to ask 
before visiting schools^ suggestions for preplanning a visitation, and 
questions to^ask during a school visit. 

Chapter 3 outlines the "tools" 'needed to evaluate programs. 
Discussed here are the philosophy, goals, and objectives of evalu- 
ation, a sample board policy on Curriculum evaluation, a list of 
tools (test scores, observations, feedback, state frame worki) for 
evaluation; and the criteria used for program evaluation. A final 
chapter describes'^how four school Boards undertook their evalua- 
tion responsibilities. • , ■ ^ . 

California State Department of Education. Elemen-^ 
cDJ • • tary Program Review Handboo/c. Sacramento: CSDE, 
. ^981. 208 pages. ED. 21 2 080. 

This handbook is designed to assist members of elementary pro- 
gram review teams in their evaluation efforts. In practice, the ma- 
jority of program reviews in California are conducted by repre- 
sentatives of the State Department of Education and by consortia dT' 
school districts. Thus this handbook is geared for use by evaluators 
at this level. But it can also be used profitably by parents or staff 
who participate in internal or self-reviews. 

The handbook is divided into three sections. The first defines a 
program review and describes the review process. Specific guide- 
lines are described for prepanng and conducting a program 
review. Including such actions as reading theVhool plan, con- 
tacti ng the school, meet ing ot h er members of the review team, o b- 
serving classrooms, reviewing records, conducting interviews with • 
individual teachers and adrninistrators, and conducting group 
int«rviews v^th the school staff and school-site cqmmittees. 

The second section goes into greater detail on how to conduct a 
program review. Three specific areas of evaluation are addressed: 
the ^ect of instruction on students, the effect of support services^ 
on instruction, and the effect of the improvement process on both 
support and instruction. For each area, specific and plentiful advice 



' is rendered on the criteria useid to judge quality, the process of 
collecting infdrmation, and the preparation of suggestions for pro- ^ 

•graro improvemtMit, The llnal section of thi^ hahdl>4>ok includes in- 
strucj^ions for preparing the report of f)r6grani.revievv findings -and 
guidelines for shaVing.the findings with the^chool community. 

• * 1^1!^^' JoseR^ M. "Student InterviewJu as an Es^alua- 
tiCT||||ol/' Kducaiionaf LeadersHlp/39, 3 (Decern-" 
ber 1^)/ pp.- 1 84-1 86. Ei;256 40j. » 

'Tor. a direct, honest assessment of what's happening in the 
•classrooms oC your school," Farley suggests, "ask the kids." Stu- 
dents are "an indisputable source ot expertise"-on all facets of the 
instructional program. Moreovel, they* can be "brutally honest." 
Tapping this rich source of evalCiative data, though, takes som^ 
common-sense human communication skills, which Farley here 
explains. * .• ; 

Students should b^andomly selected, or should be selected to • 
represent all major subgroups of the student "-population. Inter- 
viewers should explain to students, clearly and directly, why they 
are' asking questions or taking notes*. "Successful iViteryi ewers," 
says Farley/ "explaip that they are evaluating'a school or program' 
in the hope of identifying strengths as well as opportunities and are 
not- judging indivlduaUeacheTs." ' 

Elementary- students require action-oriented questions such as 
"What are you doing? Who tojd you to do it? What hapfDens if you 
do it wrong?" and so forth. Secondary students can provide Infor- 
• mation at more abstract levels and can discuss "their Individual ex- 
''periences, opinions, and feelings regarding school." 

The California State^ Department of Education has already 
developed a student-centered evaluative process that uses studegt , 
inten/iews in cOmf)ination with other information to assess school 
programs. Student interviewers in California are advised to use 
single ideas vy,hen questioi\ng students and to use "neutral ques- 
tions'' that don't lead students to the expected answer. "Students 
; are skilled in determr'ning expected responses/' Farley reminds us. 
"lnterviewers.7should not reflect biases or the right answer injlh^ir 
questioning." 

\ . Haiadyna^ Tom, "Two Approaches to Criterion-/ 
, Refer.ence.d Program Assessment." Educational 

Leadership^' 39, 6 (March 1982), pp. 467-470. Ej 259 
538. 

o : 

Many school districts rely on standardized tests to provide data 
for program evaluation: Bat these tests, says Haladyna, "frequently 
are not directly relevant to local goals and objectives.''' A better 
method of obtaining high quality diagnostic data is to' use one of 
the two criterion-referenced approaches to program evaluation 
described here: the "random sampling" plan or the "item 
response thecfry." • 

, _Both approaches require three initial steps. First, a specific set of 
curriculum objectives must be defined. If, for example, the pro- 
gram of interest is a reading program for grades 1-6, "the objectives 
should describe each and every major behavior studefjjls must ac- 
quire to become satisfactory readers by the district's standards." 

Second, the clistrict mu$t identify or create test, items gear^d-to ^^ 
each' objective. Haladyna suggests the items be collected from 
such sources as the Northwest Evaluatibn Association, a consor- 
tium^bf school districts that has developed a test itertj "bank." The 
items in tl^ls-bank-are-keVed4o.instf^Gt4onal^bj€G^^ 
already been 'field tested. If a- district is using items from other 
sources, they should go on to step three-^the item 'review and 
validation process. * 7 ' 

In the random sampling plan, th^district establishes a pool of 
.twenty to fifty validated test items for each instructional objective. 
If, for example, the district has forty objectives and wants to treate 
^^Jve different test forms, they randomly select five test Iterhs froni 
J\Jiach objective's pool and use them to create five different forty- 



Item tQSts. The test for^ns are then randc4mly distributed to all-* 
students in the fall and spring. Because sampling was random,, 
Haladyna contends, the district gains unbiased data about each in- 
structional objec!iv^ , ^' ^ ^ ^ 

Item fesponse theory is essentially similar to the random samp- 
ling plan, except that ijtems ire selected based on their difficulty for 
Individual students, and a nwrie sophisticated statistical analysis of 
the'results is used. Either approach, Haladyna concludes, "will, 
hel^ districts focus more specifically Ori instructional p/oblems art?f 
their resolution, thereby making the testing program a major cori- 
'triButor to/improv^d instruction." 

^ Iwanicki, Edward F. "Developing a Secondary 
' * School Evaluation Program.*' I^ASSP Bulletin, 60, 
^ 401 (S eptember 1 976), ^p. 71-76. Ej 1 53 .057^. 

"Evaluations If poorly conceived," says Iwanicki, "can be detri-w 
mental to the effective functioning of the secondary school." By 
using the practical, clearly stated acjvice offered here by Iwanicki, 
though, evaluation can be what it's supposed to.be— an integral 
part qf 'the school's instructional improvement effort. • 

An essential prerequisite for the development of an effective 
evaluation program is a clear statement of goals and objectives for 
the school. Iwanicki suggests'that this statement be "pyramidal" in 
structure: at the top should be general school goals" followed by 
curriculum objectives relatecfto each school goal, followed by spe^ 
dfic instruCtiontal objectives related to each curriculum objective. 
Several resources, which Iwanicki briefly describes, are available 
to help schools identify their goals and objectives. * c 
: In the -second step of program 'evaluation, the principal must 
decide what* fevel of (he pyramid an evaluation should focus on 
and what typ^s of information need to be collected at each- level. 
Then^ approf^riate evaluation instrurhents must be selected ac- 
cording to their ascuracy in measuring the objectives of the pro- 
gram being ev'hluated and according to "the. convenience with 
tvhich the results provided by the instrument car5 be used to make 
decisions" about the pFog ram under consTdefatiorK — - 

Effective program evaluation irivofves more than stating-goals 
and objectives, determining types* of feedback, and selecting 
evaluation instruments, Iwanicki concludes. It also involves "the 
integration of these somewhat technical activities into the human 
organization* of the school." Iwanicki closes with a few suggestions.* 
for facilitating this integration process. \ 

Landoh, Gienda L., and SWirer/ William. Program 
Evaluation Handbook. Madison: Wisconsin School 
Evaluation Conso rtium, 1981 . 62 pages. ED 211 576. 

Do present scheduling practices in your school or district assist 
in the process of meeting instructional objectives? Are the^levels of 
the materials in yojjr reading program appropriate for the grades 
and course^ln which they are used? What evidence exists to sup- 
port your judgment in these areas? 

These are the kinds of questions this manual poses for members 
of program evaluation committees to help them rationally think 
through complex program evaluation issues, Landon and Shirer 
have designied this manual for the final stage of an organized 
evaluation process. They assume that the dfita and other evidence 
neieded for instructionar program evaluation' h'aye already been 
collected'. The central purpose of this manual, then. Is to facilitate 
-the-final^eci-sion=naaking^ prQce s s . of pro g r a m eva l u 

: In th*e first section of this manual,*the ajjthors pose questions on 
program intent, including questions on learner expectati6ns and 
on the district's "mission" statement. For example, they ask whe- 
ther the sequence of ex(5ectations provides for appropriate rein- 
forcement of skills Snd attitudes, and whether the program is ad- 
dressing specific portions of the district's philosophy or mission. , 

A second sectlqn includes numerous questions on program de- 
livery- in particular oh such subjects as instructional materials, cur- 



» l-icufum leadership and administfatioh, instructional strategies, and 
» support program jarticuiation. Affinal section covers program out- 
. comes and incluifles thought-proyoking questions on performance 
criteria,, grades, fet results, and ^tudent and staff attitxides. 

' ' : -4 1^ '. I ^ > 

ro^ ' Libdleyij'Melinda; artd Carter, Kathy. *"An Out- 

.{b) — -c^me-Based Mod^l for School Evaluation ahu Pro-- 

'gfam Development.'-' Paper presented at the 66th 
Annual Meeting of the American .Educational Re- 
starch Association, New York, March 19-23^ 1901. 
il pages. ED 218^315. ^ ^ ' - 

-TKe Learner-Bcised Accountability System (LBAS) is^a recently 
developed model for sthool evaluation and program development 
that is nbw ip use in 174 Texas school districts.^ln this paper/ 
Lfndley and (garter describe the eleven components 'of the LBAS 
. and explain l^ow they work to translate program t<?st data into prac- 
tical classroofn applications. . ' , 

Components one through three are preparatory to the opera-, 
tional implehientatlon of the IbAS. Component one is the, all-.^ 
important firocess of developing desirable learner outcomes, 
which serve/ aj the district's or program's goals. The authors devote 
considerable attentipn to this topic and outline a five-step process 
,of developing, analyzing, and customizing program goals. Com^ 
ponents tvJo and three i'nvolye developing objectives-referenced 
tests and working out the logistics pi test administration. 

Compohents foOr»through six include the actual adrhinistratfcn 
of the tests and th(^ic scoring and reporting with the aid oN^pecially 
- • >designed comput^er prpg''3ms. Lindley and Cairter discjjss eight 
'special activities involved J n the .evajuating phase;offhe LBAS 
(components two t^irob^ph si>^), such as matching jtest items to local 
objectiveis, validatingvtest items; and providing orientation for test- 
"administf-ators. V • ^ • ' '\ 

Components seven t)?|^ough eleven deal with'^the utilization of 
the evaluation data*: The performance of individual students can.be 
" display^d^ and a^ropriateVemediai* action taken, or the overall. 
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scores fdr particular objectives can be presented for classrooms, 
schools, or the entire district. The latter means of'analysis helps 
* isolate the^ institutional* or curriculum factors associated with " 
learner neeas. ' * - ^ 



Lbs Angeles Unified School District. Prograi^-Re' 

I at^ Evaluation (Manual and. Staff Development 
. Package). Los Angeles: Research and Evaluation . 
Branch, LAUSD, July 1980.'86 pages.'EP 21 Q 292. . 

Effective program ev^*luation requires that the evaluators— whe- 
ther principals, central offic^ personnel, or other*' school-site per- 
sonnel—be well versed^ in th^ methodologies and instruments 
a*vaila6le for program evaluation. This, publication is specifically 
designed. for use in a training session for these administrators aHd 
contains all the materials ne^ssary for such a session. It ^ddress^s 
three key areas of program -evaluation: the actual assessment and 
reporting of local schools' instructional programs, the state require- 
ments regarding ohgoingevaluattoh, and the functionlngof evalua- 
tion committees: * J '** \ ^ 

The first section contains a potpourri of evaluation forms, along^ 
with direaions for their use and relatea materials. Inclucled are a \ 
^'Class Progress Form/' a "School Projgress Summary Form/' a 
"Mayagement" Review Record of Program Outcomes, " and an - 
"E\^uation Committee Time task Calendar." ^- ^ \. 

^ , Th§ second seQtion is a detailed speaker's script for an eyalud- ; 
tipn training, workshop, (n^he script, thq. evaluation forms in the ' 
first* sectio'n are moce fully explained. Al^ discussed in the script 1 
^re theoDurposes of evaluation, the importance, of foljowup, and \ 
actual evaluation techniques. " , \ • 

A third section contains a set of transparency masters for use ' 
during the training session. - J^. , 



m 



Pklahoma §tate Department of Education. Qur- 
rl&ulun) Review Handboo\<i: Language 'Arts. 
i98h1^^r Oklahoma City: Curriculum Division, 



OSDE, J981. 33 pages: ED 208 540. 



Thi^ curriculum .review handbook isjiesigned to provide edu- 
cators with a general system for ev^iluating their language ajts pro- 
grams>from kindergarten through twelfth grade. It. contains nu- 
merous questions, °evaluationforrns, checklists, and other tools ^ 
useful for characterizing and evaluating such programs. 
' Prior to conducting the- actual evaluation of tVie school's pro-^ 
gram7^e"e>^ruatof^F^^ the philosophy ahd rationale of 

the language arts program, the school's expectations regardinastu- = 
dent achievement, and the school's existing educational progranfi. 
Three separate questionnaires pos^ questions in these areas to help 
^valuators clarify the goals of evaluation and collect data needed in 
the evaluative process. . , ' • 

The largest part of this handbook is an extensive rating scale of 
various features of the language arts program. Items to be rated 
deal vyith philosophy, curr.iculurn development, facilities and ma-- 
terials, teacher involvement, and student Involvement. -Specific 
levels of the program addressed are kindergarten, lower , ele- 
mentary, upper elementary, middle school, and senior high. 
^ For example, *thife items under curriculum development ask 
whether a current'language arts curriculum guide is available, 
whether a well-organized Inservice proferam-for improving lan- 
guage arts instruction exists, and whether {he' school has a stated 
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laftguage t» r (s sequence of sk i l l s. The evaluatpr. rat e s each item on a 
scale of four (strongly agree) fo zero (strongly disagree), sum- 
.marizes the scores for each area, and identifies areas bf strength 
and areas needing attention. This handbook is one in a series (ED 
'*208 536— ED 208 542) published by the OklaVioma State Dep^art- 
'ment of Education.' Curriculum areas covered in the other hand- 
bo^oks include social studies, health education, business education, 
reading/early childhood education, and, physical education. *' 



Sungaib, Htlen. "A New Mbdel' for Program 
Evdlua<ion: Tbe 'Appreciative' One." The Australian 
Admimslrator.. ], 6 (December 1980). 6 pages. ED 
213 137. 



f^^ost models of program evaluation attempt to deternaine the 

^degree to which programs ar^ meejting predefined objectives or " 
criteria These models use aavariety of instruments and observation 
techniques to -g^yn objective knowledge about programs. All these 
models, Sungaila states, "are founded in the logical-empirical re- 
search tradition" that developed in the nineteenth century; they 
are designed to allow th^ evaluator "to cpnduct and evaluate the 
• educational affairs'for which-he or she is responsible in a rational 
fa^ion." ' ■ » 

This all sounds quite reasonable, of course, but Sung'ajla be- 
lieves that the "logical-empirical philosophy^.' of evaluation is^Vout- 
-•dated." In this article, wViich is steeped in the jargorl of the hurtian 
potential movement, she proposes an alternative— an experience- , 
base4 model of program evaluation designed to develop "an exis- 
tential understanding of the program, as ej^perienced by.the par- 
ticipants." - • ' ' 

In the first step @f this "appreciative" modef of evaluation, the ^ 
evaluator makes a videotape of the program in progress, with par- 
ticular attention^ paid to capturing "moments" of "life in.the pro-, ' 
gram," Then the evaluator views the tape with the participants and' 
intervie^vs them at length^about how they felt at these influerrtial' 
moments. Questions might include: "Did you feel there was any- 
thing getting in the way of the development of the program?" and 
"AUhat 'morfient' what was your understanding of where the pro- 
gram was going?" ' 

Using this model, Sungaila concludes, "the evaluator does not 
Jcod^ the program reality in terms of any predetermined classifi- 
cato^ scheme. Rather, he allows the pro^i^-'participants to 
'decode^, for hjefi whatibe living reality of the program was for 
them.'/*Admiij^strators who still cling to the "outdated" rational- 
empirical tradition may find Sungaila's ideas hard ta swallow, but 
there is at least some food for thought in her radical hdman poten"^ 
tfal approach. 



Way/ loyce W. A Manua/ tor Small Program Evalua- 
C±3 tion. Cortland, New York: institute for Experimenta- 
. • ' tion in Teacher Education, New York State Univer- 
. sity, February 1 981 . 46 pages. ^D .204 377.. : 

^"^3ll programs, as defined by this pianual, are those that affect . 

^ lOO^pr less students, a>e taught by five or less instructors, and. haVe 
no or limited special funding. For.example, says Way, a .small ppo-' 
gram might include a numSer of cfassfooms in tjne same lahguage 
arts program, a team of teachers using the sanje methods, or, ai its 
simplest level,- one ieacher teaching a group ^f students In a par- 
ticular area or in a particular way. V ^ 

This manual is specifically designed for the evaluation of sucH 
small programs. It assumes that the potential evaluators want to jm- 
.♦jprove their progr^s, yet have no.training in evaluatioh techniques 
and only a^small amount of money for evaluation expenses. ' 

The'fifst "task" in evaluating a program is to decide what to 
evaluate. Some possible approaches to evaluation include com- 
paring the program to alternatives, determining cost-effectiveness, 
^ and^ssessing the social and'polltical effects of jhe prGgrai^i. These 
approaches, says Way, are^difficult for a small program evafliatbr^o 
implement. A more manageable approach is to determine wWthfer 
thre program has achieved .its goals. - 

This brings us to the second task of evaluation-defining goals 
and objectives. Uslr^g extensive examples from one small program. 
Way illustrates how a reali^ic set of goals and objectives can be 
f established. Next, an evaluation technique mu^ be determined for • 
each objective. Way suggests that tomm^cial instruments and 
standardized t^sts be used wherever appropriate. If satisfactory 
instruments a^e not available, evajuators should embark bn task 
fpur-developing needed instruments. To help in this task, Way ' ' 
giyes extensive instructions bn developing tests of variotis sorts. • " . 

After reviewing the overall evaluaticJn design (task five), the 
evaluator- should use the chosen instruments to collect the data 
(task six). Again, Way gives plenty of practical advice and illus- 
trations to help inexperienced program ev.aluatQrs. TKe final three \ 
tasks Way discusses are summarizing the results, writing the ' 
evaluation report, and, finally, making decisions about the pro- 
gram based on the evaluation results. ^ 
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